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Dynamics  of  Brain  Organization 


Principles  of  Brain  Functioning 

JG  Taylor  &  S  Kasderides 
King’s  College,  London. 


There  are  presently  attempts  to  build  increasingly  global  models  of  brain  function.  This  involves  both 
the  use  of  neuranatomy  and  neurophysiology,  and  leads  to  increasingly  large  simulations.  In  this  talk 
some  of  the  principles  will  be  attempted  to  be  extracted  from  these  simulations  and  more  general 
theoretical  analysis  of  brain  models  and  functions  made  so  as  to  determine  the  general  nature  of  brain 
processing.  The  talk  will  consider  the  posterior  and  anterior  brain  regions  as  involved  in  processes  of 
global  competition  governed  by  the  two  systems  of : 

•  a)  the  thalamus-NRT-cortex 

•  b)  the  frontal-basal  ganglia-thalamus. 

The  modes  of  action  of  these  will  be  outlined  as  : 

•  (i)  concerned  with  running  a  global  competition 

•  (ii)  controlling  actions  taken  on  internal  representations  set  up  on  active  memory  which  may  lead 
to  new  representations  or  to  externally  directed  actions. 

The  first  of  these  systems  has  been  analysed  extensively,  and  allows  for  an  analysis  in  terms  of  a 
negative  Laplacian  net.  The  second  goes  under  the  heading  of  the  ACTION  network,  and  will  be 
considered  form  the  simulation  and  theoretical  points  of  view.  In  particular  the  attractor  structure  and  the 
nature  of  flows  and  mouldings  of  these  attractors  will  be  described. 

The  manner  in  which  the  ACTION  network  can  support  attention,  active  memory,  template  matching 
and  other  frontal  features  will  be  explained.  At  the  other  end  of  the  spectrum  is  the  nature  of  the  neurons 
used,  and  the  manner  they  will  affect  the  global  processing.  This  is  particularly  important  if  there  are 
chaotic  features,  as  seem  to  occur  in  EEG  patterns  in  various  cortical  regions.  The  second  part  of  the  talk 
will  describe  the  manner  in  which  compartmental  neurons  can  modify  neuronal  dynamics,  especially 
during  learning.  Different  topologies  and  connectivity  patterns  will  be  used  to  investigate  the  importance 
of  intrinsic  delays  introduced  by  the  compartmental  neurons.  Possioble  catastrophic  transitions  in  the 
parameter  space  will  be  described  as  related  to  interesting  topological  structures. 


RST:  a  connectionist  architecture  to  deal  with 
spatiotemporal  relationships 


J.-C.  CHAPPELIER 

ENST  -  Dept  INF,  46  rue  Barrault,  75634  PARIS  Cedex  13 
Fax:  (33  1)45.81.31.19 
Email;  chaps@inf .  enst .  fr 


keywords  :  Spatiotemporal  processing,  temporal  neural  networks,  “Integrate  and  Fire”  neurons,  motion  detection. 


In  the  last  decade  connectionism  has  proven  its  efficiency  in  the  field  of  pattern  recognition.  The  next  challenge  is 
to  do  as  well  for  spatiotemporal  problems.  Rather  than  improving  any  already  known  neural-network  model,  the  aim 
of  our  work  was  to  design  and  experiment  an  new  connectionist  architecture,  biologically  inspired  and  having  abilities 
for  spatiotemporal  processing  [2].  This  model  called  RST  is  based  on  the  two  following  principles: 

-  taldng  spatial  relationships  (e.g.  between  neighboring  pixels  in  an  image)  into  account  at  the  network  level, 

-  using  a  neuron  model  with  temporal  capabilities  coming  from  biology. 


Network 

Concerning  the  spatial  aspect,  the  network  is  placed  in  actual  space  (two  or  three  dimensions),  the  metrics  of  which 
directly  influence  its  structure  through  a  connection  distribution  function.  Following  the  arguments  of  Thorpe  claiming 
that  the  information  processing  in  precortical  visual  system  should  mainly  be  “feed-forward”  [4]  we  choose  for  RST 
a  “feed-forward”  network. 

A  given  number  of  neurons  is  randomly  (with  an  uniform  distribution)  chosen  among  a  portion  of  the  considered 
space.  Links  are  then  created  upon  them  according  to  their  neighborhood:  each  neuron  has  a  probability  to  be  connected 
to  a  given  previous  neuron  which  is  expressed  in  terms  of  the  polar  coordinates  of  the  considered  neuron  with  respect 
to  this  previous  neuron.  We  choose  to  implement  a  generalization  of  layered  models  but  keeping  neighborhood  rela- 
nonships  through  a  local  connectivity.  This  local  aspect  is  parameterized  by  an  angle  a  outside  which  no  connection 
IS  possible.  The  layered  aspect  is  obtained  by  introducing  a  pick  in  the  probability  distribution.  This  pick  corresponds 
to  the  average  gap  between  layers.  Notice  that  within  this  framework,  a  MLP  network  can  be  obtained  by  choosing  a 
equal  to  j  and  the  standard  deviation  of  connection  probability  much  smaller  than  its  mean  (so  as  to  get  actual  layers). 

For  practical  reasons  the  portion  of  space  among  which  the  neurons  are  randomly  set  is  delimited  by  two  segments 
(rectangle  areas  in  the  3D  case)  called  “input”  and  “output”  layers. 

Each  link  between  any  two  neurons  is  randomly  weighted  using  an  uniform  distribution  on  a  given  interval 
^maxj-  If  u^min  is  Icss  than  zcFo,  we  get  inhibitory  links  in  proportion  — 


Neuron 


The  temporal  aspect  in  RST  is  implemented  at  the  neuron  level  [3].  Because  it  is  close  to  biological  models  easy 
to  implement  and  fast  to  compute,  we  choose  an  “Integrate  and  Fire”  neuron  model  with  external  refractory  period  and 
post-synaptic  potentials  (PSPs)  modeling  (see  [5]  for  instance). 


This  model  is  basically  described  by  two  variables:  a  membrane  potential  V  and  a  threshold  ff.  V  is  actually  the 
sum  of  a  specific  potential  U  and  the  inputs  sent  by  the  connected  neurons.  Most  of  the  time  V  is  less  than  B,  but 
whenever  V  reaches  0,  the  neuron  “fires”.  It  sends  a  spike  to  the  rest  of  the  network  it  is  connected  to,  and  changes  its 
state  as  follows:  6  is  increased  by  an  amount  Afl  and  the  specific  potential  U  is  lowered  down  to  a  post-spike  value 
XJq.  When  the  neuron  does  not  fire,  the  two  variables  decay  exponentially  to  their  resting  values  Uoo  and  Boo  with  some 
time  constant  tu  and  r«. 

In  addition  to  U  and  6,  another  variable  s  is  also  introduced  in  order  to  reproduce  the  refractory  period  of  the  neuron. 
It  is  most  of  the  time  equal  to  1  and  drops  down  to  0  whenever  the  neuron  fires.  This  so-called  “shunt”  function  modu¬ 
lates  the  input  received  by  the  neuron,  so  that  its  membrane  potential  V  is  precisely:  V(t)  =  U (t)  -|-  5(i)  •  E{t).  After 
the  neuron  firing,  the  shunt  function  also  decays  exponentially  towards  its  resting  value  1  with  the  same  time  constant 
as  U. 

The  input  received  by  a  neuron  is  the  sum  over  space  (all  the  inputs  of  the  neuron)  and  time  (all  the  firing  instants 
of  those  inputs).  We  also  include  a  model  of  synaptic  transmission:  the  spikes  sent  by  inputs  of  a  neuron  are  actually 
received  as  PSPs  (having  a  larger  time  scale  than  spikes). 


Applications 

RST  has  been  applied  to  the  spatiotemporal  problem  of  motion  detection  in  sequences  of  video  images  [1].  Con¬ 
secutive  images  of  a  video  sequence  of  people  walking  in  front  of  static  objects  (e.g.  doors)  were  presented  one  after 
another  to  RST  input  layer.  The  network  parameters  were  set  (once  for  all,  without  learning)  so  as  to  filter  moving 
objects  from  static  ones.  The  only  firing  neurons  in  the  output  layer  were  those  corresponding  to  the  moving  objects 
in  the  input  layer.  The  most  important  parameters  of  the  architecture  for  this  application  appeared  to  be  the  mean  of 
connection  weights,  the  connection  angle  a  and  the  standard  deviation  of  the  connection  probability. 

By  this  application  and  other  experimental  tests,  RST  has  shown  its  straightforward  ability  — without  learning  — 
for  spatiotemporal  pattern  processing.  It  should  be  emphasized  that,  because  of  its  underlying  principles,  the  output  of 
RST  directly  has  some  topological  meaning.  There  is  no  need  for  an  external  device  to  use  the  output  in  a  spatiotemporal 
topological  manner. 
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This  work  comprises  three  stages. 

1)  First  a  recurrent  cluster  of  spiking  neurons  is  analysed  qualitatively.  These  neurons  have  a 
probabilistic  synaptic  transmission  and  EPSC  are  extended  in  time  and  are  modelled  by  alpha  functions. 
Investigated  clusters  have  typically  100  excitatory  neurons,  with  random  connections  to  10%  of  the 
other  neurons  in  the  cluster,  and  to  neurons  in  an  input  layer.  The  cluster  as  a  whole  responds  with  a 

g  gam  to  specific  input  patterns  and  enters  in  an  almost  self-  sustained  firing  mode.  This  response  is 
c  aracterized  by  a  jitter  in  starting  time,  a  latency,  and  a  progressive  decay  of  the  activity  when  the  input 
pattern  is  removed  These  properties  can  qualitatively  be  reproduced  by  an  abstract  short-term  memory 
(STM)  neuron  performing  the  coincidence  detection  of  input  spikes  and  exhibiting  sustained  firing. 

2)  A  feedforward  network  of  these  STM  neurons  is  analysed  theoretically  and  by  simulation.  Analytical 

1QQ4?  T  ®  ‘o  input  jitter  and  level  of  activity  [Bugmann  and  Taylor, 

99  ].  The  network  IS  provided  with  a  feedback  reset  scheme  which  interrupts  the  sustained  firing  in 
lower  layers  once  Sm  neurons  in  higher  layers  have  recognised  their  input  patterns,  and  are  thus 
entering  a  sustained  firing  mode.  Such  a  network  can  be  used  to  model  visual  masking,  where  it  predicts 
prolonged  sustained  finng  in  lower  layers,  as  observed  by  Rolls  and  Tovee  [1994].  The  feedback 
VI  “^d'^ces  an  oscillatory  activity  in  the  network,  and  predicts  oscillations  in  the  LGN  controlled  by 
[Rktoond  S  d  ^19^7^  ^  post-stimulus  histograms  are  observed,  as  in 

3)  It  is  explored,  via  a  simulated  model,  how  these  dynamical  properties  can  be  are  exploited  in  the 
design  of  a  model  for  recognition  driven  visual  saccade  control.  The  model  is  a  2-stream  artificial  vision 

biological  vision  system.  Several  biological  observations  are  at  the  basis  of  the 
system.  -  In  area  IT  of  the  monkey,  neurons  are  found  which  respond  specifically  to  certain  objects  but 
independently  on  the  position  of  the  object  in  the  visual  field  [Chelazzi  et  al,  1993].  This  means  i)  that 
the  object  recognition  process  in  the  brain  is  location  invariant  and  that  a  model  such  as  the  one 

Th  ^  identifies  objects  but  does  not  tell 

here  they  are.  ^ereby,  another  system,  for  localisation,  must  be  coupled  to  the  IT  stream  -  All  the 

neurons  m  area  IT  respond  initially  to  their  preferred  object  in  the  visual  field,  but  if  there  is  a  specific 
object  towards  which  the  neuron  should  look,  then  only  the  neurons  coiresponding  to  that  objecLtay 
active,  all  others  becoming  silent  [Chelazzi  et  al,  1993].  This  tells  us  that  tLe  is  an  attentional  procLs 
which  can  control  the  activity  of  neurons  in  IT.  However,  a  neuron  being  active  does  still  not  tell  where 
the  object  IS,  so  as  to  guide  the  eye  movements.  -  Saccadic  eye  movements  towards  a  point  in  visual 
space  are  controlled  by  neurons  m  the  Superior  Colliculus  (SC).  The  model  proposed  here  exploits  the 
synchronization  occumng  in  lower  layers,  due  to  the  recognition  process  in  higher  layers,  to  tag  object 


locations  in  a  way  that  can  then  be  picked  up  by  the  parietal  stream  and/or  eventually  the  SC  to  drive  eye 
saccades. 
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Temporal  aspects  of  cortical  processing  in  primate  visual  cortex 


Jean  Bullier 

Cerveau  et  Vision  INSERM  U371 
69675  Bron  Cedex 


It  is  usually  assumed  that  information  in  the  primate  visual  system  is  processed  in  serial  or  hierarchical 
fashion.  Neurons  of  the  lateral  geniculate  nucleus  (LGN)  which  relay  the  retinal  signals  project  almost 
exclusively  to  area  17,  also  called  area  VI.  Area  VI  constitutes  the  lowest  level  of  a  hierarchy  of  cortical 
areas  of  which  areas  V2,  V3,  V4  and  MT  constitute  the  successively  higher  processing  stages.  It  is 
known  that  when  area  VI  is  inactivated  or  lesioned,  other  cortical  areas,  such  as  V2,  V3,  V4,  MT  are 
silenced  or  their  activity  is  strongly  reduced.  The  serial  model  suggests  that  information  is  first 
processed  within  area  VI,  transferred  to  area  V2  for  further  processing  before  being  sent  to  areas  MT  or 
V4. 

If  this  was  the  case,  neuronal  activity  corresponding  to  a  visual  stimulus  would  travel  through  the  visual 
cortex  as  a  wave  starting  in  area  VI  and  ending  in  the  temporal  and  parietal  cortices.  Such  a  model  of 
organization  poses  serious  problems  for  the  temporal  integration  of  neural  activity  between  neurons  of 
the  areas  located  at  different  stages  of  the  hierarchy.  Because  of  the  slow  rate  of  information  transfer  in 
axons  of  the  cerebral  cortex,  feedback  signals  would  come  too  late  to  act  upon  the  incoming  wave  of 
activity.  Another  potential  problem  is  related  to  the  slow  integration  times  of  cortical  neurons.  Visual 
recognition  can  be  done  in  150  msec.  Neuronal  signals  arrive  in  area  VI  30  msec  after  the  stimulus.  If 
one  estimate  the  average  integration  time  of  cortical  neurons  at  10  msec  and  the  average  conduction  time 
between  areas  at  5  msec,  one  is  left  with  8  as  the  maximum  number  of  processing  stages  before  visual 
recognition  is  achieved,  i.e.  1  neuron  per  level  of  the  hierarchy  (Thorpe  and  Imbert  1989) 

Because  of  such  difficulties  associated  with  the  serial  model  we  have  tested  its  plausibility  by  recording 
in  areas  VI  and  V2  which  constitute  the  first  two  stages  of  the  cortical  hierarchy.  We  measured  the 
latencies  of  the  responses  of  cortical  neurons  in  areas  VI  and  V2  to  flashes  of  light  centered  in  their 
receptive  fields  to  determine  whether  the  responses  of  neurons  in  area  V2  lag  behind  those  of  neurons  in 
VI.  The  latencies  of  neurons  in  areas  VI  and  V2  are  spread  across  a  wide  range  (30  to  150  msec)  and 
the  latencies  to  visual  stimulation  of  V2  neurons  lag  by  about  10  msec  behind  those  of  neurons  in  VI 
(Nowak  et  al.  1995) .  A  substantial  part  of  this  latency  difference  (5  msec)  can  be  accounted  for  by 
conduction  times  in  the  axons  between  the  two  areas.  If  one  assumes  that  cortical  processing  time  is 
reflected  in  the  latencies  of  cortical  neurones,  our  data  suggest  that  such  processing  is  done  in  parallel  in 
the  two  areas  and  a  serial  model  appears  very  unlikely.  This  is  further  supported  by  the  results  of 
cross-correlation  experiments  showing  that  many  neurons  in  areas  VI  and  V2  synchronize  their  firing 
and  do  not  display  characteristic  patterns  of  neurons  in  area  VI  driving  neurons  in  area  V2,  as  expected 
from  a  serial  model  of  organization. 

Comparison  of  published  data  in  areas  VI,  V2  and  MT  also  show  that  neuronal  activity  begins  at  the 
same  time  in  areas  V2  and  MT,  despite  the  fact  that  area  MT  is  placed  three  levels  higher  than  V2  in  the 
hierarchy  of  cortical  areas  (Bullier  and  Nowak  1995) .  Such  data  suggest  that  neural  activity  is 
simultaneous  in  most  cortical  areas  beyond  VI. 

We  have  also  investigated  whether  neural  activation  resembles  a  wave  spreading  through  areas  VI  and 


V2.  It  is  well  established  that  visual  information  is  relayed  to  the  cortex  of  area  VI  through  two  major 
channels,  the  M  channel  circulating  through  the  magnocellular  layers  and  the  P  channel  through  the 
parvocellular  layers  of  the  lateral  geniculate  nucleus.  A  certain  amount  of  functional  specialization  is 
found  among  these  channels  :  color  information  is  carried  almost  exclusively  by  the  P  channel  whereas 
high  temporal  frequency  and  low  contrast  information  is  transferred  through  the  M  channel.  However, 
nothing  is  known  concerning  the  way  they  interact  during  visual  perception.  We  have  addressed  this 
question  by  measuring  latencies  to  visual  stimulation  at  different  sites  within  areas  VI  and  V2  that  are 
driven  by  the  M  and  P  pathways. 


We  found  that  neurons  of  area  V 1  driven  by  the  M  channel  are  activated  20  ms  earlier  than  those  of  the 
P  channels  (Nowak  et  al.  1995) .  This  difference  is  important  since  it  is  twice  the  latency  difference 
between  successive  stages  in  the  hierarchy  of  cortical  areas,  like  between  areas  VI  and  V2. 

In  V2,  neurons  in  the  thick  cytochrome  oxidase  bands,  that  belong  exclusively  to  the  M  channel,  and 
neurons  of  the  pale  bands,  that  are  driven  by  M  and  P  inputs,  are  activated  20  ms  earlier  than  neurons  of 
the  thin  cytochrome  oxidase  bands  (Munk  et  al.  1995) .  The  consequence  of  the  early  latencies  in  the 
thick  cytochrome  oxidase  bands  is  that  their  target,  area  MT,  and  the  parietal  cortex  are  rapidly  activated 
by  visual  stimuli.  Interesting  speculations  can  be  made  of  the  latency  difference  between  thin  and  pale 
cytochrome  oxidase  bands  in  areas  VI  and  V2.  Neurons  of  the  thin  bands  respond  to  low  spatial 
frequencies,  are  often  color-coded  and  can  be  thought  of  coding  the  surfaces  of  objects  whereas  neurons 
of  the  pale  bands  may  be  involved  in  coding  the  orientations  and  positions  of  edges.  Since  both  the  thin 
Md  pale  bands  project  to  V4  which  acts  as  a  relay  in  the  ventral  occipito-temporal  stream  that  is 
involved  in  object  recognition,  our  results  suggest  that  information  about  edges  of  objects  are  transferred 
faster  to  temporal  cortex  than  information  about  surfaces.  The  early  firing  of  area  V2  by  neurons  of  the 
M  pathway  coding  for  edges  may  be  also  important  in  triggering  activation  of  the  feedback  pathway 
from  V2  to  VI  that  could  mediate  interactions  in  VI  with  the  delayed  P  information  concerning 
surfaces. 

What  are  the  functional  consequences  of  this  difference  in  timing  between  the  M  and  P  channels  at  later 
stages  of  the  visual  cortex  ?  Neurons  of  area  V4  receive  information  from  neurons  of  the  M  and  of  the  P 
channels,  information  that  is  transfered  to  the  inferotemporal  cortex  which  is  important  for  pattern 
recognibon.  Modem  hypotheses  suggest  that  pattern  recognition  cannot  be  properly  achieved  by  a 
purely  bottom-up"  approach.  Instead,  it  is  supposed  to  involve  a  bidirectional  process.  A  sketchy 
representation  of  a  pattern  activates  the  inferotemporal  cortex  that  sends  back  signals  through  feedback 
connections  corresponding  to  hypotheses  concerning  the  representation  of  the  pattern.  This  descending 
information  is  supposed  to  interact  with  incoming  information  arriving  in  extrastriate  cortical  areas 
Such  a  scheme  requires  the  presence  of  a  system  that  provides  the  early  sketchy  description  of  the 
pattern.  We  propose  that  this  corresponds  to  the  neural  activity  carried  by  the  M  pathway  into  the 
inferotemporal  cortex. 
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Some  recent  experiments  from  our  laboratory  (Thorpe,  Fize  &  Marlot,  in  press)  have  used  Event  Related 
Potentials  to  measure  how  long  it  takes  for  the  human  visual  system  to  process  complex  natural  images. 
Subjects  had  to  decide  whether  a  briefly  presented  colour  photograph  (presentation  time  20  ms)  contains 
an  animal  or  not.  Target  images  included  a  very  large  range  of  animals  in  their  natural  habitat 
(mammals,  birds,  reptiles,  fish...)  whereas  the  distractors  included  a  wide  variety  of  natural  scenes 
(forests,  lakes,  mountains,  flowers  and  fruit...).  Despite  the  very  severe  challenge  to  the  visual  system 
posed  by  such  a  task  (each  image  was  only  ever  seen  once,  thus  ruling  out  image  specific  learning) 
subjects  performed  remarkably  well,  averaging  94%  correct  with  reaction  times  that  varied  from  380  to 
580  ms.  Even  more  surprising  was  the  discovery  that  scalp  potentials  recording  from  frontal  sites 
revealed  a  clear  differential  effect  that  distinguished  between  animal  and  non-animal  stimuli  at  about 
150  ms  after  stimulus  onset,  implying  that  the  visual  processing  necessary  to  perform  this  task  has 
already  been  achieved  at  this  time. 

Given  the  anatomical  and  physiological  constraints  imposed  by  our  knowledge  of  the  visual  system, 
such  data  have  major  implications  for  theories  of  visual  computation.  In  particular,  they  imply  that  the 
processing  necessary  for  this  task  can  be  done  on  the  basis  of  a  feed-forward  pass  through  the  visual 
system.  Furthermore,  given  the  large  number  of  layers  of  neurones  involved  it  would  appear  that 
conventional  firing  rate  codes  would  be  too  slow.  An  alternative  hypothesis  treats  individual  neurones  as 
analog  to  delay  converters.  In  such  a  case,  rapid  processing  could  be  obtained  by  using  the  order  in 
which  neurones  generate  spikes  as  a  code.  Recent  work  indicates  that  this  sort  of  spike  order  coding 
could  well  provide  an  important  key  to  understanding  the  remarkably  efficiency  of  the  human  visual 
system. 
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The  mammalian  neocortex  is  subdivided  into  a  number  of  functionnal  areas.  Each  of  them  is  connected 
to  other  cortical  areas,  and  receive  information  from  the  sensory  surface.  Experimental  studies  have 
shown  that  cortical  neurons  activated  by  external  stimulus  could  synchronize  their  spike  trains.  One 
important  question  concerning  cortical  synchronization  is  to  determine  whether  it  is  mediated  by 
thalamic  inputs  (which  receive  information  from  the  sensory  surface)  or  cortico-cortical  connections 
(which  receive  information  from  other  cortical  area  s). 

A  network  architecture  inspired  from  biology  has  been  built.  The  neural  net  is  composed  of  4  layers  of 
neurons.  The  first  area  models  part  of  the  retina,  that  receives  images  of  the  external  w  orld.  Layer  2 
models  an  intermediate  area  between  the  retina  and  cortical  layers  which  receives  informations  from 
retina  (LGN).  Two  opposite  layers  3L  and  3R  model  parts  of  Left  and  Right  hemispheres  in  the  visual 
cortex.  This  network  corresponds  to  a  monocular  vision.  Our  network,  like  most  biological  networks,  is 
not  fully  connected;  the  synaptic  organization  of  the  network  is  initially  imposed,  by  biological 
constraints.  The  spatial  distribution  of  connections  is  randomly  chosen  among  layers,  using  u  niform 
distributions. 

A  probabilistic  model  is  proposed  to  model  the  behaviour  of  neurons.  This  model  takes  into  account 
both  spatial  and  temporal  integrations  with  intern  eural  delays.  Its  own  particularity  is  the  model  of  the 
refractory  period  using  for  each  neuron,  a  spike  memory  term,  which  sum  up  the  spiking  history  of  the 
neuron  itself.  Neuron  states  are  binary  {0,1 },  and  dynamic  is  stochastic. 

Simulating  a  visual  environment  of  moving  bars  on  "the  retina"  leads,  in  our  artificial  neural  network,  to 
the  emergence  of  synchronization  between  neurons  of  opposite  layers,  even  if  these  neurons  have  been 
receiving  information  from  others  layers  with  different  temporal  delays.  Mutually  coupled  oscillator 
cells  belonging  to  different  opposite  layers  emerge  too. 

Our  simulations  reproduce  results  of  biological  experiments  which  observe  synchronization  and 
oscillations  between  cells  of  the  two  oppposite  hemispheres,  activated  by  external  stimulus.  During 
these  simulations,  special  connectivity  between  cells  has  shown  to  be  an  important  parameter  for  the 
emergence  of  synchronization  but  the  spike  memory  term  introduced  in  our  model  is  an  important  factor 
wich  induces  synchronization  between  cells  in  a  neural  network  of  spiking  cells. 

Different  models  have  already  been  proposed  in  the  litterature  to  explain  synchronization  or  oscillations 
in  the  visual  cortex. 

In  our  model,  oscillations  between  cells  are  not  a  consequence  of  the  intrinsinc  periodicity  of  particular 
neurons  such  as  pacemaker  neurons  and  synchronization  between  cells  depends  not  only  on  well  chosen 
delays  between  cells,  but  mostly  on  an  internal  parameter  of  spike  memory. 
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Abstract 

A  continuous  time,  on-line  training  algorithm  is  presented,  which  learns  to  classify 
input  patterns  P  in  a  Euchdean  space  R'"  (or  a  Hilbert  space)  by  attractors  of  a 
differential  equation  dx/dt  =  F{x)  in  (or  a  Banach  space).  These  nets  include 
simple  one- layer  adeline-hke  nets  as  well  as  many  recurrent  nets.  The  attractors  can 
be  quite  arbitrary — fixed  points,  hmit  cycles,  fractals,  etc. —  but  the  training  algorithm 
requires  that  we  know  approximately  where  they  are. 

A  time- varying  d  x  m  memory  matrix  M  feeds  an  input  pattern  P  of  norm  1  into 
the  activation  level  x  =  MP  €  R<*.  We  assume  both  x  and  M  change  simultaneously, 
according  to  differential  equations  of  the  following  form; 

(y-g(MP))P^, 

-y  +  MP  +  I(t) 

where  g  is  the  (vector)  transfer  function,  and  I{t)  is  the  training  input,  which  may  be 
turned  on  or  off  from  time  to  time. 

Then  x,y  satisfy  the  system 

dx 

n  = 

dy  ,  ^ 

^Tt  =  -!'  +  9W- 

This  is  independent  of  P,  but  the  meaning  of  x  depends  on  P.  For  sufficiently  small 
time  constant  r  >  0,  the  behavior  of  x  is  well  approximated  by: 

dx  /  ^  X 

—  =  -x-\-g{x)=F{x). 

It  turns  out  that  any  finite  set  of  patterns  Pa  can  be  reliably  associated  to  given 
attractors  Aa  for  ()  provided  that: 


dM 

dt 

dt 
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•  the  Pa  are  sufficiently  close  to  being  orthogonal; 

•  r  is  sufficiently  small; 

•  each  pattern  is  presented  sufficiently  frequently; 

•  the  training  signal  presented  at  least  once  with  Pa  takes  values  in  a  convex  set 
contained  in  the  basin  of  attraction  of  Aa- 
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Analog  Computation  and  Dynamical  Systems 

Hava  T.  Siegelmann 


This  talk  describes  a  new  interface  between  dynamical  systems  and  analog  computers.  As  opposed  to 
previous  models  we  do  not  apply  some  nature  principles  to  the  Turing  model  but  rather  start  from 
nature-based,  realizable  chaotic  dynamical  systems  and  interpret  their  evolution  as  generalized 
computation. 

By  applying  the  basic  computational  terms  —  e.g.,  halting,  computation  under  resource  constraints, 
nondeterministic  and  stochastic  dynamics  —  to  dynamical  systems,  a  nature-based  computational  theory 
will  be  developed. 


Learning  Mechanisms  in  Cognitive  Systems 
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The  usual  classification  problem  for  which  most  of  the  neural  networks  have  been  designed  is  then  to 
find  synaptic  matrices  with  which  the  network  maps  inputs  of  a  prescribed  sequence  of  patterns  to 
outputs  ,  through  one,  several  or  a  continuous  set  of  layers  .  One  can  use  several  results  on  control  of 
nonlinear  systems  to  obtain  learning  algorithms  converging  to  some  solutions  of  this  problem,  such  as 
the  back-propagation  formula  (which  is  nothing  other  than  the  adjoint  equation  in  control  theory). 
The  mathematical  structure  of  the  space  of  synaptic  matrices  involving  tensor  products,  we  then  obtain 
in  the  formulas  describing  the  learning  algorithms  pure  tensor  products  ,  whose  entries  are  the 
products  of  the  presynaptic  and  postsynaptic  activities.  This  explains  in  a  mathematical  way  the 
emergence  of  Hebbian  rules  in  learning  mechanisms  of  neural  networks. 

However,  by  looking  only  at  static  or  asymptotic  problems  (mapping  inputs  onto  outputs,  or  finding 
equilibria  and  more  generally,  attractors),  the  evolution  mechanism  is  neglected  and  the  “real”  time 
used  to  model  the  neural  networks  is  often  replaced  by  artificial  times.  For  instance,  the  algorithmic 
time  ,  describing  the  iterations  of  a  given  algorithm,  is  often  interpreted  as  a  learning  rule,  although  it 
may  not  be  related  to  the  time  involved  in  the  modeling  of  the  network.  Or,  in  the  case  of  multi-layer 
networks,  the  layers  of  the  network  represent  a  second  time  scale. 

By  choosing  the  route  used  by  most  neural  networks,  one  may  bypass  the  basic  question:  Why  and  How 
do  synaptic  matrices  evolve?  and  the  basic  answer:  To  adapt  to  viability  constraints  through 
learning  laws  which  are  feedbacks  of  the  neural  network  regarded  as  a  control  system,  associating  at 
each  instant  with  any  signal  a  synaptic  matrix  allowing  the  adaptation  to  viability  constraints. 


In  order  to  highlight  this  problem  of  adaptation  to  viability  constraints,  one  can  enrich  the  concept  of 
neural  network  by  embedding  it  into  the  class  of  “cognitive  systems”  which  should  be  closer  to  some 
cognitive  considerations  than  pure  neural  networks. 

The  aim  of  a  cognitive  system  is  to  adapt  to  (viability)  constraints  imposed  by  the  environment  through 
a  recognition  mechanism”,  and  “action”  and  “perception”  laws  regulated  by  “conceptual  controls” 
(which  can  be  synaptic  matrices). 


The  variables  of  the  cognitive  system  are  described  by  its  state  and  a  regulatory  control,  which  we  call 

conceptual  control.  The  state  of  the  system  (henceforth  called  the  “  sensori-motor  state”)  is  described 
by: 

*  the  state  and  the  variations  of  the  environment  on  which  the  cognitive  system  acts. 

*  the  state  of  cerebral  motor  activity  of  the  cognitive  system,  which  guides  an  individual’s  action  on  the 
environment. 


The  regulatory  control  of  the  cognitive  system  is  described  by  an  endogenous  cerebral  activity  which 


is  not  genetically  programmed,  but  acquired  by  learning  and  recorded  in  the  memory  .  The 

purpose  of  this  activity  is  to  “interpret”  (or  “illuminate”)  the  sensory  perception  of  the  environment. 

A  learning  rule  is  then  a  set-valued  map  associating  to  each  sensori-motor  state  of  the  cognitive  system  a 
set  of  regulatory  controls  allowing  to  adapt  to  the  viability  constraints. 

One  can  characterize  and  study  the  learning  mechanisms  which  allow  the  cognitive  system  to  adapt  to 
viability  constraints,  to  prove  the  existence  of  a  largest  learning  mechanism,  and  to  build  specific 
“heavy  evolutions”  obeying  an  “inertia  principle”. 
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Abstract 

Networks  of  spiking  neurons  can  realize  pattern  recognition  tasks  by  encoding  the  patterns 
in  the  synaptic  strength  of  connections  between  neurons.  We  examine  a  system  that  receives 
as  an  input  a  linear  combination  of  its  stored  patterns  and  outputs  the  coefficients  of  the 
linear  combination.  Both  the  input  and  the  output  are  given  in  temporal  coding,  hence  the 
computation  is  based  on  the  explicit  firing  times  of  the  input  neurons  and  not  on  the  more 
common  firing  rate  of  neurons. 

The  precise  formulation  imposes  several  constraints  on  the  system,  which  can  be  only 
approximately  fulfilled  by  biological  neurons.  By  simulations  we  show  that  the  above  described 
approach  can  be  nevertheless  realized  by  biological  neurons  in  a  very  reliable  way,  giving  rise  to 
a  very  simple  and  straightforward  method  for  pattern  recognition  in  biological  neural  systems. 

Various  types  of  neural  networks  are  known  to  be  suitable  for  pattern  recognition  tasks.  Most 
of  them  work  on  the  basis  of  firing  rates  and  not  of  single  firing  events.  However  there  exists  sub¬ 
stantial  evidence  that  biological  neural  systems  can  perform  certain  computations  on  the  basis  of  a 
few  spikes.  It  is  also  known  that  the  timing  of  such  spikes  can  be  very  precise  (see  e.g.  [Mainen  95]]. 
We  focus  here  on  a  system  where  the  computation  is  based  on  single  firing  events.  The  inputs 
and  outputs  are  represented  as  time-differences  between  firing  times  of  certain  neurons.  One  can 
distinguish  two  different  approaches,  depending  on  whether  one  stores  the  patterns  in  the  delays 
([Hopfield  95])  or  in  the  weights  of  certain  synapses.  We  focus  here  on  the  latter  one,  since  in  this 
model  querying  a  mixture  of  those  patterns  and  also  learning  appear  to  be  easier  explainable.  The 
basic  construction  has  been  presented  in  [Maass  95].  It  is  based  on  a  two  layer  network  consisting 
of  simple  integrate-and-fire  neurons.  Our  goal  is  to  adopt  this  approach  to  biological  realistic 
neurons  and  to  investigate  how  reliable  these  neurons  (with  different  types  of  noise,  non-linear 
effects  in  the  synaptic  integration  etc.)  perform  on  this  approach. 

The  basic  idea  of  this  model  can  be  described  ais  follows:  Each  nemron  of  the  first  layer  forms 
some  synaptic  connection  to  each  neuron  of  the  second  layer.  The  system  has  stored  m  patterns, 
each  consisting  of  n  components.  It  has  n  neurons  in  the  first  layer  (the  input  neurons)  and 
m  neurons  in  the  second  layer  (the  output  neurons).  The  patterns  are  stored  in  the  weights  of 
the  synaptic  connections  between  the  two  layers  such  that  the  weight  for  the  connection  from 
the  ith  input  neuron  to  the  jth  output  neuron  represents  the  ith  component  of  the  jth  stored 
pattern.  The  system  receives  as  input  linear  combinations  of  the  stored  patterns  in  temporal 
coding,  i.e.  the  time  difference  between  the  firing  time  of  the  ith  input  neuron  and  the  time,  when 
some  “reference” -spike  is  produced,  represents  the  ith  component  of  the  input  vector.  In  the 
same  fashion  the  output  is  encoded  by  making  all  output  neurons  fire,  where  the  time  difference 
between  the  firing  of  the  jth  output  neuron  and  the  reference  spike  represents  the  strength  of 
the  jth  pattern  in  the  input.  We  assume  the  existence  of  an  additional  input  neuron  uq,  which 
produces  the  reference-spike.  In  order  to  achieve  the  desired  firing  times  of  the  output  neurons, 
the  following  assumptions  are  made: 

•  the  stored  patterns  have  to  be  from  some  bounded  interval:  x°‘  6  [-K,K]^,a  —  l,...,m 
for  some  proper  K  €  R'*'  and  have  to  be  orthogonal,  i.e.  =  0  for  a  /3 


•  the  initial  segment  of  the  initial  rising  part  of  the  EPSP  (respectively  decreasing  part  in  the 
case  of  an  IPSP)  can  be  described  by  some  linear  function.  PSP’s  (postsynaptic  potentials) 
describe  the  influence  of  the  firing  of  the  presynaptic  neuron  on  the  postsynaptic  one.  An 
excitatory  (inhibitory)  PSP  increases  (decreases)  the  postsynaptic  potential  thus  making  it 
more  (less)  likely  that  the  postsynaptic  neuron  fires. 

•  the  time  difference  between  the  firing  time  of  any  input  neuron  and  the  reference  spike  may 
not  exceed  a  certain  predetermined  value. 

•  the  weights  of  the  connections  firom  the  input  neuron  uq  to  the  output  layer  are  chosen  such 
that  for  each  output  neuron  the  sum  over  the  weights  of  all  incoming  connections  equals  a 
certain  predetermined  constant.  This  guarantees  that  all  output  neurons  will  fire,  such  that 
the  time  difference  between  the  firing  times  of  the  output  neurons  and  the  reference  spike 
represents  the  output. 

This  model  has  several  essential  advantages:  The  whole  computation  can  be  performed  by 
biological  neurons  in  a  very  short  time  (approximately  10  ms).  Since  the  output  is  encoded  in 
the  same  fashion  as  the  input  one  could  imagine  a  sequence  of  such  systems,  which  first  recognize 
low-order  patterns  and  later  high-order  patterns.  Hence  complex  recognition  tasks  can  be  solved 
by  systems  m  a  time  range  which  is  known  to  sufiice  also  for  various  neural  system  of  animals  and 
humans.  In  this  model  it  is  also  very  easy  to  process  sequences  of  inputs,  which  makes  it  easy  to 
examine  the  time  dependency  of  the  coefficients  of  the  linear  combination. 

In  order  to  find  out  how  strongly  this  model  depends  on  the  abovementioned  assumptions,  we 
are  making  computer  simulations  with  the  neural  simulation  system  GENESIS  (see  [Bower  941).  In 
addition  we  want  to  find  out  how  biological  neurons  that  are  more  realistic  than  simple  integrate- 
and-fire  neurons  perform  on  this  approach.  The  output  neurons  were  modelled  by  several  com¬ 
partments  such  that  each  synaptic  input  to  some  output  neuron  is  on  a  different  branch  of  the 
dendritm  tree  of  that  output  neuron.  The  advantage  of  remote  synapses  is  that  the  linear  range 
for  the  interaction  is  much  wider  than  for  synapses  on  one  branch  (this  means  that  the  influence 
of  the  various  PSP’s  at  a  certain  time  on  the  potential  of  the  soma  simply  sum  up).  Despite  of 
the  fact  that  PSP’s  m  biological  neurons  do  not  contain  linear  segments  they  approximate  a  linear 
function  for  a  short  time  (approximately  3  ms)  very  well.  The  scaling  of  the  weights  is  a  crucial 
point  in  our  model,  since  large  weights  cause  non-linear  effects  in  the  PSP’s.  If  the  weights  are 
too  small,  then  noise  on  the  membrane  potential  or  on  the  threshold  will  have  a  stronger  influ- 
ence^  We  were  able  to  show  that  the  system  behaves  within  quite  a  wide  range  very  stable  and 
reliable  despite  of  those  constraints  especially  for  the  case  that  all  stored  patterns  consist  only  of 
positive  reals.  In  the  case  of  negative  components  IPSP’s  must  be  used.  Since  the  efiiciency  of  an 

mhibitory  synapse  is  usually  lower  than  that  of  an  excitatory  one,  the  range  of  possible  weights 
is  smaller  here. 

Finally  we  observe  that  it  is  not  necessary  that  the  orthogonality-condition  for  the  stored 
patterns  is  precisely  fullfilled.  If  for  example  some  related  patterns  are  not  pairwise  orthogonal 
(e.g.  X  -x  >  0  for  related  patterns  x°‘  and  x^)  then  the  presence  of  one  of  those  patterns  enhances 
the  output  for  the  other  of  those  patterns,  which  appears  to  be  also  the  case  in  certain  biological 
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It  became  apparent  in  the  mid-1980’s  that  none  of  the  approaches  to  a  successful  automatic  target  recognition  (ATR) 
system  were  working.  The  fundamental  reason  was  that  no  one  had  any  provable  method,  or  algorithm,  that  could 
demonstrably  handle  the  problem.  A  huge  amount  of  time,  effort,  and  money  was  put  into  ATR,  yet  the  fundamental 
problem  remained  unsolved.  The  three  steps  to  ATR  were  well  known:  1.  Segment  the  image  into  regions.  2.  Compute 
features  from  the  image  regions.  3.  Classify  the  regions  hy  their  features.  This  is  elementary  stuff  that  is  done  by  living 
creatures  of  the  most  stupid  variety,  yet  it  eluded  the  legions  of  frustrated  researchers  for  decades.  Flys  do  it  Fish  do 
it  Spiders  do  it,  even  little  spiders. 

The  problem  was  that  the  basic  algorithm  did  not  exist  At  least,  that  is,  not  in  the  research  literature.  It  definitely 
existed  in  Nature.  It  was  reasonable  to  go  back  and  study  what  Nature  did,  find  out  what  segmentation  actually  was, 
what  features  were,  and  not  only  how  the  classification  worked  but  exactly  what  was  classified  in  the  first  place.  By 
discovering  the  concepts  of  the  processing  found  in  Nature,  one  could  (if  successful)  then  throw  out  the  biological 
means  and  keep  the  method,  l.e.,  the  algorithm.  So  the  idea  was  to  look  for  the  algorithm  as  it  is  found  in  Nature. 

What  could  be  found  in  neurodynamical  systems  that  would  apply  to  the  ATR  algorithm? 

The  simplest  neurodynamical  system  is  a  single  neuron.  The  neural  input  from  its  synpatic  fields  is  a  time  signal  of 
filtered  pulses  at  various  and  variable  rates.  It  feeds  an  integrate-  and  fire  pulse  generator  to  make  the  output  pulse 
train  to  other  neurons.  The  neurons’  ieaky-capacitor  structure  of  both  the  main  body  as  well  as  the  dendrites  provides 
some  very  complex  interactions  among  the  input  signals.  The  original  pulse-coupled  neural  network  is  the  Eckhom 
linking  field  model.  Its  essential  core  is  that  the  inputs  come  in  two  separate  internal  channels,  not  one,  and  that  these 
two  channels  multiply  each  other  to  form  the  final  voltage  potential  that  drove  the  pulse  generator. 

Each  neuron  ordinarily  generated  a  pulse  train  whose  pulse  rate  increased  with  the  strength  of  the  synaptic  field 
input.  However,  due  to  the  second  channel  (also  with  its  own  synaptic  field),  when  both  channels  were  active  the 
neuron  could  be  forced  to  trigger  its  pulses  in  approximate  synchronization  with  the  nuerons  feeding  it.  The  neurons 
would  then  link  up  in  the  sense  that  they  would  all  fire  together  in  synchronous  bursts.  Since  this  could  only  happen  if 
the  neurons  had  roughly  the  same  levels  of  input  strengths,  only  those  with  similar  inputs  (think  intensity  grey  levels 
in  an  image)  would  link  up.  The  network  performed  grouping  by  similarity.  This  was  a  fundamental  new  thing  in 
neural  net  theory. 

It  turned  out  that  the  artifice  of  requiring  two  channels  and  a  tiny  multiplier  somewhere  in  the  cell  was  not  necessary. 
The  standard  cell  model  can  do  it  But  to  show  this  required  the  development  of  a  new  mathematical  tool  for  functions 
of  a  matrix  that  let  the  function  itself  be  written  as  yet  another  matrix.  It  only  applies  to  small  matrices,  which  in  turn 
means  it  can  only  solve  a  very  simple  cell  geometry.  (It  has  to  be  a  single  cell  with  two  inputs,  and  in  addition  the 
inputs  had  to  be  constant  in  time.)  But  it  did  allow  inclusion  of  the  integrate-and-fire  pulse  generator  (which  made  it  a 
coupled  nonlinear  pair  of  differential  equations)  and  a  good  model  of  the  synapses  themselves,  the  shunting  model. 

The  bottom  line  is  an  algorithm.  Its  parts  are: 

1.  A  pulse-generator,  to  transform  an  intensity  image  into  a  binary  pulse  image.  The  pulse  rates  increased  with  the 
intensity. 


2.  Pulse  coupling,  so  neighboring  cells  could  synch  up  if  they  pulsed  at  similar  rates,  i.e.,  were  dirven  by  similar 
intensities. 


3.  Different  celluiar  configurations,  causing  each  ceil  type  to  have  it  own  pulse  pattern,  or  rhythm,  impressed  on  its 
output  pulse  train  and  convoived  with  its  inputs  as  well. 

4.  Pulse  signal  combination,  done  by  the  ceil  as  a  complicated  function  involving  weighted  products  of  ali  the  input 
time  signal  from  other  celis  and  groups  of  cells. 


All  four  parts  are  done  by  the  same  small,  simple,  nonadaptive  circuit  The  circuit,  well  within  the  capability  of 
current  electronic  chip  technology,  could  then  be  made  as  the  core  of  a  smart  pixel  array  of  arbitrary  size,  and  turned 
out  to  be  extremeiy  fast  The  first  crude  proof  of  concept  chip,  with  eight  celis  in  a  row,  ran  at  1  megahertz.  It  was 
hard  to  make  these  chips  run  slow  enough;  they  perferred  the  megahertzes.  They  still  had  a  good  dynamic  range  still 
could  perform  their  four  tasks,  and  fit  well  with  image  applications.  ' 


Although  only  bits  and  pieces  of  the  overall  system  have  been  investigated  at  this  time,  it  is  possible  to  look  ahead  and 
see  how  it  can  fit  together.  Here  is  how  it  could  work:  An  input  image  is  transformed  into  a  pulsing  image.  SlmUar 
pixels  group,  similar  groups  also  group  (the  PCNN  can  segment  not  only  intensity  regions  but  also  extended  intensity 
padient  regions),  and  each  object  in  the  scene  generates  its  own  characteristic  time  signal.  These  time  signals  are 
invariant  against  all  the  known  geometrical  changes  due  to  the  camera-scene  geometry;  translation,  rotation,  scale 
persperiive  and  out-of-plane  rotations  (distortions),  and  overall  scene  illumination.  Each  color,  or  spectra,  is  sensed 
by  a  different  pixel  layer  as  in  a  standard  color  video  camera,  but  in  the  PCNN  each  color  type  has  a  slightly  different 
pixel  geometry.  For  example,  the  capacitances  can  be  different  for  each  color  type.  The  the  objects’  time  signals  in 
each  color  are  d^tinct  This  is  found  to  be  true  in  the  human  retinal  gangilion  cells,  which  have  different  shapes  for 
ea^  color  type.  The  signals  in  each  color  for  each  object  are  multiplied,  fusing  the  time  signals  of  the  objects’  pattern, 
texture  shape  and  spectral  hue  into  a  singie  composite  time  signal.  The  signal  goes  to  an  adaptive  classffier  where  it  is 
correiated  with  a  previously  impressed  reference  signal,  and  the  level  of  the  correlation  serves  to  identify  the  object 


F^sed  nets  are  intrinsically  indifferent  to  space  and  time.  They  work  in  both  domains  equaliy  well  because  (when  the 
ultimate  rasue  w  identified)  time  causality  and  linking  are  due  to  the  decay  of  the  signals  in  the  elementary  capacitors 
of  the  ceils,  whde  spatial  linking  is  due  to  the  locai  dendritic  interconnects  among  the  cells. 


What  prevents  the  PCNN  from  being  a  full-up  automatic  target  recognition  system  is  that  it  needs  something  to 
compare  its  outputs  to,  a  reference  signal  with  which  to  correlate.  This  is  the  next  phase  of  the  research.  Since  the 
standard  nonpulsed  neural  net  theories  have  for  the  most  part  been  deeply  concerned  with  these  adaptive 
classification  techniques  for  decades,  and  since  the  PCNN  is  completely  compatible  with  and  complementary  to  these 
adaptive  models,  there  is  a  large  body  of  knowledge  available  for  guidance  in  the  next  phase  of  this  work.  There  are 
other  methods,  too,  and  if  they  are  more  appropriate  there  is  no  reason  not  to  use  them. 


A  large  ^ortoent  of  adaptive  techniques  are  available.  For  example,  the  total  time  signal  of  an  object  can  be 
re-broadcast  back  into  the  linking  wave  patterns  making  the  components  of  the  time  signal  in  the  first  place,  and 
a  aptiyely  encoded  into  the  waves.  This  puts  information  about  the  whole  object  into  each  of  its  parts,  as  is  done 
op  ically  in  holograms.  Alternatively,  different  objects  can  encode  each  other’s  time  signals  in  their  linking  wave 
pattern  so  that  the  presence  of  one  can  recall  the  other.  Or,  more  simply,  the  entire  neural  adaptive  process  can  be 
discarded  and  a  digital  time  signal  reference  used  instead. 


There  are  good  near-term  benefits  in  the  segmentation  properties  of  PCNN’s,  and  in  the  mid  to  long  term  the  new 
aJgonthm  has  strong  promise  of  solving  fully  the  problem  of  how  to  make  an  automatic  target  recognition  system  that 
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Cortical  maps  reflect  a  precise  topographic  neuronal  organization  where  neighboring  sites  of  maximum 
relative  activity  on  the  cortical  plane  correspond  to  nearby  points  in  parameter  space.  Because  in  such 
cortical  structures  signal  processing  relies  entirely  on  patterns  of  connectivity,  cortical  maps  have  been 
suggested  to  be  computationally  efficient.  This  fact  is  supported  by  recent  quantitative  studies  performed 
in  topographically  organized  primate  visual  areas,  that  is  in  VI,  in  the  inferotemporal  (IT)  cortex,  and  in 
the  superior  temporal  sulcus  (STS),  where  cells  discriminate  different  complex  visual  stimuli  within  5  to 
10  ms.  Because  firing  rate  has  been  shown  to  be  rarely  in  excess  of  1  spike  in  a  5-10  ms  period  in  cortex, 
one  of  the  key  question  is  whether  cortical  maps  can  operate  on  the  basis  of  single  spikes  in  multiple 
input  sources. 

The  neural  operation  particularly  at  stake  within  this  context  is  the  competition  among  neurons  of  the 
map  for  maximal  response  to  a  given  stimulus.  Such  a  competitive  mechanism  is  of  prime  importance 
for  adjusting  the  response  properties  of  cortical  units.  Its  realization  in  terms  of  single  spikes  has  been 
formally  expressed  and  shown  to  rely  on  a  modulation  of  individual  excitatory  post-synaptic  potentials 
(epsp)  using  a  mechanism  that  should  not  affect  the  somatic  membrane  potentid.  Moreover,  and  to 
account  for  fast  cortical  processing,  we  advocate  an  inhibitory  modulation  so  that  input  signals  coming 
first  are  left  nearly  unaltered,  while  those  coming  later  are  much  more  attenuated,  giving  more 
importance  to  signals  with  short  latencies.  Candidate  mechanisms  for  such  a  modulation  are  presynaptic 
inhibition  and  shunting  inhibition,  both  of  which  has  been  described  and  suggested  to  operate  in  the 
central  nervous  system.  Realization  of  this  modulation  takes  advantage  of  a  simplified  normalization  of 
the  neuronal  signals,  an  operation  usually  defined  in  terms  of  Euclidean  norms,  and  thus  difficult  to 
justify  biologically.  Computer  simulations  of  this  architecture  yielded  processing  times  as  short  as  4  ms 
for  the  correct  activation  of  a  cortical  map  comprising  1000  neurons. 
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We  discuss  different  levels  of  information  processing  in  cerebral  cortex.  We  show  that  information  is 
embedded  in  the  interspike  intervals  of  single  cells.  There  is  also  information  content  in  the  interaction 
between  two  neurons.  However,  it  is  only  at  network  level  that  pattern  discrimination  and  motion 
detection  become  possible. 

We  present  a  model  cortex  made  of  two  interconnected  layers.  Each  layer  is  a  network  of  oscillators  in 
interaction,  the  networks  are  in  chaotic  state.  In  this  state  the  model  cortex  can  not  process  information. 
However,  if  unstable  orbits  embedded  in  chaotic  attractors  are  stabilized,  the  system  may  categorize 
patterns  and  detect  motion. 
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Continuously  processing  sensory  signals  and  generating  behavioral  responses,  the  brain  is  capable  of  self-sustained 
activity  patterns  across  ensembies  of  neurons.  These  ensemble  patterns  allow  for  the  generation  of  oscillations,  such  as 
neural  patterns  involved  in  walking  and  swimming  rhythms,  and  are  invoived  in  recognition  of  sensed  patterns  as 
they  occur.  In  addition,  the  brain  can  think  and  imagine  for  chosen  periods  of  time.  This  extensive  set  of  temporal 
capabilities  is  supported  by  a  neural  system  architecture  with  an  inherent  ability  to  operate  in  the  temporal  domain 
and  to  generate  self-  sustained  activity  patterns  for  prolonged  periods  of  time.  We  would  ultimately  like  to  capture  the 
cogent  temporal  dynamics  from  biological  neural  systems  in  computational  models,  with  the  goal  of  building  new 
computational  paradigms  that  are  powerful  with  respect  to  analysis  and  generation  of  temporal  signals  and  patterns. 

Dynamic  neurai  network  models  are  capable  of  producing  prolonged  self-sustained  activity  and  pattern  generation, 
with  changes  in  activations  continuing  across  a  network  over  long  periods  of  time.  Activation  patterns  can  be  stable, 
with  no  changes,  or  can  resonate  with  simple  oscUlations,  limit  cycles,  or  chaotic  oscillations.  With  the  modulation  of  a 
weight  multiplier  g,  the  dynamics  can  progress  from  simple  to  more  complex  patterns,  including  chaos  (Doyon  et  al, 
1993, 1995;  Sompolinsky  et  al,  1988).  There  is  a  rich  potential  for  computational  paradigms  that  could  take  advantage 
of  the  self-sustained  temporal  dynamics  and  the  dynamic  attractors  in  neural  networks. 

In  this  paper,  we  explore  dynamics  that  arises  in  neural  network  models  and  identify  relationships  and  patterns  that 
could  be  useful  in  computational  paradigms  that  utilize  dynamic  neural  networks.  We  present  observations  of  the 
activation  dynamics  of  single-layer  recurrent  networks,  with  random  weights,  as  they  progress  from  simple  to 
complex  activity  patterns.  We  consider  that  a  dynamic  network  paradigm  could  potentially  be  used  for  pattern 
classification,  where  different  initial  states  would  evoke  different  end-state  oscillations,  or  different  external  inputs 
would  modulate  those  oscillations,  yielding  pattem-to-oscillation  maps. 

We  have  explored  situations  in  which  an  external  stimulus  -  a  pattern  -  is  applied  to  a  choatic  network,  and  locks  the 
network  into  a  limit  cycle  attractor  ("attractor  locking")  (Palmadesso  and  Dayhoff,  1995).  Unique  patterns  then  evoke 
different  limit  cycles,  and  could  in  principal  be  distinguished  by  the  limit  cycles  that  they  evoke. 

Networks  with  multiple  dynamic  attractors  were  developed  from  networks  with  only  one  observed  dynamic  attractor, 
using  a  weight  modification  scheme  designed  to  increase  the  multiplicity  of  the  dynamic  attractors  (Dayhoff  et  al, 
1994).  The  goal  of  the  training  was  to  develop  more  than  one  dynamic  attractor  in  the  neural  network,  and  to 
constrain  the  basins  of  the  attractors  so  that  a  set  of  chosen  initial  states  would  be  forced  into  di^erent  basins.  The 
chosen  states  would  not  have  to  occur  in  the  attractor  of  its  basin.  Successful  training  was  done  on  a  variety  of 
networks,  including  a  10-neuron  network  that  was  trained  with  over  100  attractors.  After  training,  most  of  the  initial 
states  evoked  difl'erent  oscillations  in  the  network. 

Dynamic  neural  networks  have  a  tremendous  flexibility  possible  in  their  basins  of  attraction,  as  demonstrated  through 
a  study  of  binary  networks  (Dayhoff  and  Palmadesso,  1995).  We  define  the  basin  class  capacity  as  the  capacity  of  a  set 


of  neural  networks  to  exhibit  a  variety  of  different  basin  classes.  Simulations  show  that  the  basin  class  capacity  of  a 
binary  network  with  only  5  neurons  has  over  a  thousand  classes,  and  the  basin  ciass  capacity  increases  rapidly  with 
increasing  size  of  the  network.  This  high  capacity  is  computationaliy  significant  because  it  creates  flexibility  in  how 
the  basins  of  attraction  can  divide  up  all  of  the  possible  states.  This  division  could  ultimately  be  used  in  pattern 
classiHcation  or  other  computations,  where  the  attractor  reached  could  represent  an  answer.  Since  we  aim  to 
eventually  have  paradigms  that  allow  adjustment  and  training  of  attractor  basin  boundaries,  we  have  accomplished 
the  first  step  towards  this  aim,  to  explore  how  many  sets  of  basins  and  basin  boundaries  are  possible  with  weight 
adjustments. 

We  have  trained  an  a  priori  chosen  dynamic  attractor  into  networks  with  a  feed-forward  configuration  that  has  a 
single  recurrent  loop  from  output  to  input  layer  (Lin,  Dayhoff,  and  Ligomenides,  1995).  These  networks  included 
time-delays  and  adaptive  time-delay  mechanisms.  At  each  iteration,  the  network  input  was  a  segment  of  a  trajectory 
and  the  network  produced  the  values  for  the  next  position  along  the  trajectory.  The  networks  were  trained  on  circle 
and  figure-eight  trajectories,  and  were  able  to  eliminate  noise  when  a  noisy  initial  trajectory  was  supplied.  These 
capabilities  would  be  useful  in  robotic  trajectory  generation  and  in  control  applications. 

Complex  dynamics  generated  by  dynamic  networks  has  yet  to  be  fully  tapped  for  applications  and  computational 
purposes.  Important  issues  remain  to  be  resolved  about  dynamic  neural  networks  before  they  can  be  fully  understood 
and  fully  developed.  These  issues  include  (1)  how  the  weights  and  interconnection  topology  determine  the  dynamic 
self-sustained  activity  of  the  network,  (2)  the  number  of  attractors  in  the  resulting  network,  (3)  the  boundaries  of  the 
basins  of  attraction,  and  (4)  the  training  of  attractors  into  a  network.  When  these  issues  are  resolved,  we  may  then 
understand  how  the  tremendous  span  of  a  network’s  dynamic  activities  can  be  used  in  computational  paradigms 
suitable  for  pattern  recognition,  signal  analysis  and  control. 
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Chaos  has  become  a  popular  issue  in  many  different  scientific  disciplines.  From  biology  to  physics 
some  common  features  have  emerged,  showing  that  chaos  may  be  a  general  phenomenon  appearing  in 
vanous  fields  including  neural  dynamics.  Babloyantz’s  team  has  exhibited  a  chaotic  behaviour  in  some 
parts  of  the  brain.  Freeman’s  investigations  on  the  olfactory  bulb  of  the  rabbit  showed  that  its  dynamics 
was  chaotic,  and  that  recognition  of  a  learned  pattern  is  linked  to  a  dimension  reduction  of  the  dynamics 

on  a  much  simpler  attractor  (near  limit  cycle).  Both  addressed  the  problem  of  the  use  and  utility  of  chaos 
for  information  processing  purposes. 

Some  authors  have  described  chaotic  neural  networks.  These  models  range  from  some  neurons  to  larger 
models.  Besides,  theoretical  studies  on  chaos  in  (infinite-sized)  random  neural  networks  without 
learning  have  been  performed  in  particular  by  Sompolinsky,  Crisanti  &  Sommers.  By  studying  a  discrete 
ime  random  neural  network  with  assymetric  couplings,  our  team  showed  their  results  could  be  extended 
o  a  wider  class  of  neural  networks.  We  are  able  to  to  determine  the  range  of  random  weights  for  which 

the  network  IS  chaotic.  We  can  also  explain  the  various  changes  observed  in  the  dynamical  regime  when 
sending  static  random  patterns.  ® 

focused  our  study  on  the  effects  of  learning  in  chaotic  neural  networks.  We  propose  a 
Hebb-hke  le^mg  rule  to  store  a  pattern  as  a  limit  cycle  or  strange  attractor.  We  numerically  show  the 
dynamics  reduction  of  a  finite-size  chaotic  network  during  learning  and  recognition  of  pattern.  We  have 
learSng^^^^^^^  remiscient  of  Freeman’s  observations  :  we  observe  a  reduction  of  the  dynamics  by 

behaviour  observed  during  the  learning  phase  on  the  basis  of  our 

interpret  learning  as  a  motion  in  parameter  space.  This 
pace  IS  cut  into  several  bifurcation  manifolds.  The  crossing  of  these  manifolds  leads  to  drastic  changes 
n  the  dynamics^  Some  important  indications  on  their  structures  can  be  obtained,  giving  an  interesting 
inteipretation  of  learning  :  the  learning  dynamics  is  viewed  as  an  evolutionary  p^Lss^ccurri^g  ulr 
the  evolutive  influence  of  the  pattern  to  learn.  It  leads  the  system  close  to  the  "edge  of  chaos"  In  a 

sensitive  to  the  presentation  of  the  learned  pattern.  The  respond  of  the  trained 
network  is  specific  to  this  pattern. 
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Oscillatory  attractor  neural  networks  can  perform  temporal  segmentation,  i.e.  separate  the  joint  inputs 
they  receive,  through  the  formation  of  staggered  oscillations.  This  property,  which  may  be  basic  to  many 
perceptual  functions,  is  investigated  in  the  context  of  a  symmetric  dynamical  system.  The  fully 
segmented  mode  is  one  type  of  limit  cycle  which  this  system  can  develop.  It  can  be  sustained  for  only  a 
limited  number  n  of  oscillators. 

The  limitation  to  a  small  number  of  segments  is  a  basic  phenomenon  in  such  systems.  Within  our  model 
we  can  explain  it  in  terms  of  the  limited  range  of  narrow  subharmonic  solutions  of  the  single  nonlinear 
oscillator.  Moreover,  this  point  of  view  allows  us  to  understand  the  dominance  of  three  leading 
amplitudes  in  solutions  of  partial  segmentation  which  are  obtained  for  high  n.  The  latter  are  also 
abundant  when  we  replace  the  common  input  with  a  graded  one,  allowing  for  different  inputs  to 
different  oscillators. 

Switching  from  a  common  constant  input  to  one  with  fluctuating  components,  we  find  that  noise 
facilitates  segmentation.  Using  an  appropriate  noisy  input  activating  a  few  oscillators  at  a  time,  an 
unlimited  segmentation  pattern  can  be  generated. 
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Both  single-cell  and  network  level  modeling  studies  have  neurobiological  relevance.  The  combination 
of  different  modelling  strategies  will  be  demonstrated.  The  dynamics  of  the  olfactory  bulb  -OB-  (the 
firstrelay  center  of  the  olfactory  system)  is  the  subject  of  the  studies. 

The  rhythmic  and  arrythmic  behavior  of  the  OB  occurs  due  to  the  excitatory  and  inhibitory  interactions 
betweem  mitral  and  granules  cells.  Systematic  numerical  bifurcation  analysis  showed  that  chaos  occur  in 
case  of  excitatory  lateral  connections  between  mitral  cells.  The  associative  memory  character  of  the  OB 
is  also  demonstrated. 

Simulation  of  detailed  multi-compartmental  models  of  single  mitral  and  granule  cells  have  been  done 
using  morphological  and  physiological  data.  The  morphological  data  used  to  characterize  the  basic 
structure  of  each  cell  types  are:  soma  size,  length  of  interbranch  segments,  diameter  of  branches, 
branching  probabilities,  and  density  of  dendritic  spines.  Physiological  data  on  the  kinetics  of  voltage- 
and  calcium-dependent  ion  channels  are  incorporated  into  the  model.  The  rhythmic  and  bursting  firing 
patterns,  the  role  of  the  individual  ionic  currents,  signal  propagation  through  the  different  compartments 
of  the  cells  are  simulated. 

Synchronized  activities  in  networks  may  emerge  due  to  both  excitatory  and  inhibitory  coupling. 
Computer  simulations  have  been  done  by  the  NEURON  simulation  program. 

Thanks  to  Peter  Adoijan,  Ildiko  Aradi,  George  Bama  and  Tamas  Grobler  for  their  help. 
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Numerous  models  have  been  proposed  to  represent  the  collective  activity  of  a  biological  neural  network 
which  consists  in  neurons  linked  by  means  of  excitatory  or  inhibitory  connections.  In  this  paper,  we 
make  an  attempt  to  model  in  a  simple  (but  not  too  simplist)  way  an  interconnected  neural  network  which 
receives  stationnary  external  stimulations. 


This  model  was  first  studied  in  Cottrell  (1992)  in  a  special  case  where  the  connections  are  only 
inhibitory.  It  is  based  on  a  simple  single  neuron  model  called  sandglass  model. 

At  every  moment,  the  state  of  each  neuron  is  modelled  by  the  time  until  firing ,  that  is  the  time  that  is 
supposed  to  remain  until  its  next  discharge.  This  time  flows  like  a  sand  glass  until  the  spike  emission. 
The  external  stimuli  are  taken  into  account  by  the  fact  that  each  neuron  has  its  own  frequency  of  firing, 
which  is  constant  with  respect  to  the  time,  but  can  differ  from  one  neuron  to  another  one.  The  inhibitory 
(or  excitatory )  effect  suffered  by  a  neuron  when  one  of  its  neighbors  discharges  is  modelled  by  a 
positive  (or  negative)  delay  which  increases  (or  decreases)  its  state. 


he  mathematical  and  numerical  study  of  the  global  state  shows  that  for  a  given  external  stimulation,  one 
can  observe  completely  different  behaviors,  according  to  the  values  of  the  positive  or  negative  delays 
associated  with  the  inhibitory  or  excitatory  connections.  We  show  that  the  model  exhibit  two  kinds  of 
limit  state.  In  the  convergent  case,  the  system  is  ergodic  and  all  the  neurons  have  a  positive  mean  firing 
rate.  In  the  divergent  case,  it  appears  some  activity  maps  which  look  like  the  real  ones  that  can  be 
observed  in  real  situations  :  some  neurons  are  silent  while  the  others  are  active..  The  complete 
mathematical  study  was  possible  until  now  only  when  the  connections  are  only  inhibitory.  In  that  case, 
one  can  characterize  the  values  of  the  positive  delay  which  produce  a  divergent  map.  In  the  general 
setting,  we  study  the  role  of  the  parameters  which  quantify  the  strengths  of  the  connections  by  numerical 
experiments. 


In  the  last  part  of  our  paper,  we  concentrate  ourselves  on  the  role  of  lateral  inhibition  and  we  study  it  in 
the  frame  of  the  network  composed  of  the  glomeruli  in  the  olfactory  system  of  the  insect  for  example,  as 
in  Fiat  (1994).  We  show  that  our  model  allows  to  get  some  glomerular  map  quite  similar  to  the  real  ones 
which  can  be  viewed  as  the  system  responses  to  an  odor.  We  prove  that  the  role  of  the  inhibitory 
connections  is  to  increase  the  contrast  of  the  responses,and  to  improve  the  qualitative  discrimination 
power  of  the  system. 

Cottrell  M.,  (1992),  Mathematical  analysis  of  a  neural  network  with  inhibitory  coupling  Stochastic 
Processes  and  their  Applications,  40,  1 03- 1 26. 

Fiat  F.,  (1994),  Modelisation  du  systeme  olfactif  par  une  approche  neuromimetique  :  aspects  coenitifs  et 
theoriques,  Ph.D.  thesis,  Universite  Paris  1 . 
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The  high  degree  of  similarities  between  olfactory  systems  across  the  species  is  an  incentive  to  attempt  to 
build  a  generic  model  of  the  olfactory  processing.  The  olfactory  tract  can  be  split  into  three  levels:  the 
receptor  cells,  the  glomerular  stage,  where  the  signal  is  supposed  to  be  processed  (extracted  from  noise 
and  stabilized)  and  a  third  stage  where  the  long  term  memory  processing  seems  to  take  place.  The 
glomerular  structure  and  the  neuronal  typology  strongly  suggest  that  the  glomeruli  are  functional  units, 
and  it  is  hypothesized  that  their  different  activities  form  a  glomerular  image  in  response  to  the  receptor 
cell  activity  generated  by  the  applied  ’odour’.  According  to  a  recent  result  of  molecular  biology,  each 
type  of  receptor  cells  projects  mainly  into  a  single  glomerulus.  The  dominant  presence  of  an  inhibitory 
neurotransmitter  at  this  level  indicates  that  the  synapses  are  essentially  inhibitory. 

The  formal  system  that  we  analyze  is  proposed  as  a  model  of  the  glomerular  stage.  The  architecture 
chosen  is  coherent  with  the  previously  mentioned  results  from  neurobiology.  Trying  to  keep  the  model 
as  simple  posible,  we  represent  each  glomerulus  by  a  formal  neuron,  inter  and  intra  glomerular 
connections  are  inhibitory,  and  there  is  a  one-to-one  excitatory  connection  between  receptors  and 
glomeruli;  all  delays  and  synaptic  weights  are  assumed  to  be  equal;  synchronous  dynamics  is 
investigated. 

This  model  leads  to  an  in-depth  understanding  of  the  space  of  representation  of  the  stimuli;  because,  in 
the  deterministic  case,  steady-states  are  generally  not  ’fixed  points’  but  cycles,  the  question  of  the  nature 
of  the  ’glomerular  image’  in  such  a  system  arises.  The  analysis  of  the  emergent  code  suggests  an  ability 
of  the  system  to  extract  ’key  features’  in  a  complex  and  fluctuating  signal  from  receptors,  with  an 
intrinsic  robustness  with  respect  to  the  noise  present  in  the  input  signal. 

The  properties  of  the  proposed  model  can  be  summarized  as  follows: 

.  (i)  it  computes  with  attractors,  leading,  as  usual  in  such  systems,  to  a  ’contraction ’from  the  input  space 
to  the  output  space; 

.  (ii)  because  the  inputs  alter  the  energy  landscape,  a  given  input  may  lead  to  several  attractors, 
depending  on  the  state  of  the  network  at  the  time  of  application  of  the  input; .  (iii)  nevertheless,  minima 


of  the  energy  Inndscnpe  are  preserved  if  some  features  of  the  inputs  are  preserved;  as  a  consequence, 
once  the  system  has  reached  a  steady  state  after  the  application  of  a  given  input,  it  stays  in  this  state 
irrespective  of  the  fluctuations  of  the  inputs,  provided  the  latter  comply  with  specific  conditions. 

The  properties  of  the  model  will  be  investigated  an  put  into  the  perspectives  of  its  biological  relevance 
and  of  its  engineering  capabilities. 
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In  the  context  of  both  sensory  coding  and  signal  processing,  building  factorized  codes  has  been  shown 
to  be  an  efficient  stratategy.  In  a  wide  variety  of  situations,  the  signal  to  be  processed  is  a  linear  mixture 
of  statistically  independent  sources.  Building  a  factorized  code  is  then  equivalent  to  performing  blind 
source  separation.  Thanks  to  the  linear  structure  of  the  data,  this  can  be  done,  in  the  language  of  signal 
processing,  by  finding  an  appropriate  linear  filter,  or  equivalently,  in  the  language  of  neural  modeling, 
by  using  a  simple  feedforward  neural  network. 

In  this  contribution  we  discuss  several  aspects  of  the  source  separation  problem.  We  give  simple 
conditions  on  the  network  output  which,  if  satisfied,  guarantee  that  source  separation  has  been  obtained. 
Next  we  derive  new  algebraic  solutions.  Then  we  study  adaptive  approaches,  in  particular  those  based 
on  redundancy  reduction  and  maximisation  of  mutual  information.  We  show  how  the  resulting  updating 
rules  are  related  to  the  BCM  theory  of  synaptic  plasticity.  Eventually  we  shortly  discuss  extensions  to 
the  case  of  non  linear  mixtures.  In  all  this  contribution  we  will  take  care  to  put  into  perspective  our  work 
with  other  studies  on  source  separation  and  redundancy  reduction.  Details  will  be  published  elsewhere. 
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n  recent  experimental  data,  obtained  with  multi-electrode  recording  techniques,  quite  significative 
correlations  between  the  firing  times  of  one  or  several  neurons  have  been  found.  This  seems  to  indicate 
that  precise  spatio-temporal  patterns  play  an  important  role  in  the  neuronal  coding  of  information. 

The  concept  of  synfire  chains,  proposed  by  Abeles  in  order  to  explain  these  correlations,  has  given  rise 
to  a  large  number  of  numerical  studies,  essentially  based  on  simulations.  The  classical  models  of  neural 
networks  either  have  completely  artificial  dynamics  (this  is  the  case  of  spin-based  models  like  the 
Hopfield  model)  and  are  therefore  unable  to  account  for  precise  firing  timings  --  or  deal  with 
spike-rates  or  activities  —  and  do  not,  therefore,  provide  any  clue  to  spatio-temporal  firing  correlations. 

Most  studies  on  synfire  chains  are  based  on  deterministic  equations  for  the  evolution  of  the 
depolarization  potential,  while  firing  is  governed  by  a  potential-dependent  probability.  The  coupling 
with  afferent  neurons  therefore  involves  precise  firing  times  which  may  be  obtained  from  a  random 
generator.  This  kind  of  formalism  is  well  adapted  to  simulations,  but  not  at  all  to  theoretical 
developments. 


Starting  from  a  description  based  on  the  internal  states  of  a  neuron  (characterized  either  by  the 
depolarization  potential  or  as  a  refractory  state),  we  present  a  set  of  stochastic  equations  which  gives  a 
complete  description  of  the  dynamics  of  the  whole  network.  These  equations  are  based  on  rather  smooth 
assumptions  and  model  the  inter-neuronal  coupling  with  the  help  of  phenomelogical  post-synaptic 
potentials.  The  model-dependant  evolution  equation  for  the  depolarization  potential  and  the 
phenomenological  inter-neuronal  coupling  equation  are  complemented  by  a  conservation  equation  for 
occupation  probability  of  the  neuronal  state.  This  conservation  equation  is  an  extension  of  the  classical 
Fokker-Planck  equation. 

This  set  of  equations  provides  a  frame  for  the  discussion  of  neural  network  models.  It  should  also  offer  a 
safe  basis  for  theoretical  studies  on  the  stability  or  the  coupling  of  synfire  chains. 
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This  paper  describes  a  model  architecture  based  on  the  hypothesis,  first  proposed  by  Von  der  Malsburg 
et.  al  [Marlsburg86],  that  temporally  coinciding  stimuli,  in  certain  areas  of  the  neocortex,  are  represented 
by  correlated  firing  of  neurons.  The  network,  with  an  initially  fully  connected  hidden  layer,  uses 
temporal  cross-correlations  between  the  activities  of  its  hidden  layer  as  a  basis  for  mapping  similar 
activities  into  subregions  with  limited  overlap  between  different  subregions.  Each  pair  of  units  is 
coupled  by  an  oscillator  whose  function  is  to  test  for  (dis)similarity  between  the  activities  of  the  pair. 

The  oscillator  subsequently  generates  a  reciprocal  zero-one  correlation  length  that  defines  the 
instantaneous  level  of  cooperation/competition  between  the  pair.  Since  the  (dis)similarity  matrix  is 
computed  before  the  application  of  a  window-like  non-linearity,  it  can  be  stated  that  the  subspace 
spanned  by  the  active  hidden  units,  in  the  different  subregions,  corresponds  to  the  space  of  the  principal 
component  eigenvectors  of  the  input. 

A  primary  objective  is  then  to  maximize  the  correlations  between  the  activities  of  the  units  in  a 
subregion  in  order  to  preserve,  as  much  as  possible,  the  internal  structure  of  a  given  data  pattern  in  the 
high  dimensional  space  while  discounting,  to  a  certain  extent,  factors  that  may  be  irrelevant  to 
recognition.  Relevant  features  are  uncovered  by  assigning,  to  each  hidden  unit,  a  non-linear  function 
with  a  probability  density  distribution  that  slightly  deviates  from  the  normal  (Gaussian)  distribution.  The 
motivation  for  this  is  based  on  the  fact  that,  if  one  wishes  to  identify  "interesting"  features  in  some  data, 
then  it  suffices  to  look  for  directions  onto  which  projections  are  as  non-Gaussian  as  possible  [Fyfe94, 
BarlowQO].  The  complete  response  function  is  derived  from  the  observation  that  neuron  pairs  often 
subtract  their  outputs,  yielding  characteristic  window  response  functions.  The  height  and  width  of  each 
window  (kernel)  can  be  temporally  altered  by  the  value  of  the  local  (temporal)  cross-correlation 
function,  also  referred  to  as  the  short-term  memory  (STM)  of  the  hidden  unit.  The  model  neuron  uses 
the  dynamically  computed  STM  to  achieve  averaging  where  data  points  are  sparse  and  locality  where 
data  points  are  dense.  In  other  words,  each  unit  uses  its  STM  to  temporally  (de)emphasize  its  role  in 
extracting  statistically  significant  features  from  the  given  multidimensional  data.  This  operation  also 
helps  to  sift  and  eliminate  irrelevant  noise  since  only  relevant  features  are  acknowledged. 

In  order  to  counter  the  inhibitory  effect  of  its  STM,  each  unit  also  has  a  positive  feedback  signal  for 
self-amplification.  A  corresponding  intrinsic  significance  of  the  STM  is  that  of  stability.  The  resulting 
interplay  of  recurrent  inhibition  and  recurrent  excitation  (on  presentation  of  a  new  example)  allows 
subregions  to  coalesce,  disintegrate  or  fluctuate  in  size  while  simultaneously  feeding  the  (output)  layer 
above.  Both  the  input  and  output  weights  are  updated  in  proportion  to  the  average  output  response  from 
the  hidden  local  regions  defined  by  the  corresponding  kernels  (i.e.,  Hebbian  learning).  In  order  to 
eliminate  the  need  to  propagate  the  error  derivatives  from  the  output  to  the  input  layer,  each  hidden  unit 
also  attempts  to  match  its  input-driven  bottom-up  information,  from  the  input  layer,  with  its 


corresponding  vector  of  output  weights.  In  this  way,  the  output  weight  vector  biases  the  input  weight 
vector  towards  regions  of  the  input  that  cause  the  greatest  error  at  the  output,  since  the  magnitude  of  the 
output  weight  vector  is  proportional  to  its  contribution  in  reducing  this  error. 

Both  sets  of  weights  are  trained  together  on  a  pattern  by  pattern  basis.  The  input  weights  are  updated  by 
defining  an  objective  function,  one  for  each  hidden  unit,  which  is  then  minimized  through  gradient 
descent.  The  definition  of  the  hidden  (dis)similarity  matrix  through  activity  correlations  together  with 
the  Hebbian-type  (correlation-based)  update  rules  led  to  the  network  simply  being  referred  to  as  a 
correlation-based  network  (CBN).  The  completely  feedward  nature  of  the  network  renders  it  very 
attractive  for  implementation  in  parallel  hardware.  A  scaly  architecture  is  also  described  that  eliminates 
the  interconnect  complexity  in  the  hidden  layer.  The  feasibility  of  the  approach  has  been  demonstrated 
with  some  simulation  results. 
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An  important  difference  between  biological  vision  systems  and  their  electronic  counterparts  is  the  large 
number  of  feedback  signals  controlling  each  aspect  of  the  image  collection  process. 

For  every  forward  path  of  information  in  the  brain,  from  sensor  to  comprehension,  there  appears  to  be 
several  neural  bundles  which  send  information  back  to  the  sensor  to  modify  the  way  the  information  is 
collected.  In  this  paper  we  will  examine  the  role  of  such  feedback  signals  and  suggest  algorithms  for 
intelligent  processing  of  images  directly  on  the  focal  plane,  using  feedback.  We  consider  first  what  form 
these  signals  might  take  and  how  they  can  be  used  to  implement  functions  common  to  conventional 
image  processing  with  the  objective  of  moving  the  computation  out  of  the  digital  domain  and  place 
much  of  it  on  the  focal  plane,  or  analog  processing  close  to  the  focal  plane.  While  this  work  fedls  under 
the  general  heading  of  artificial  neural  networks,  it  goes  beyond  the  static  processing  of  signals 
suggested  by  the  McCulloch  and  Pitts  model  of  the  neuron  and  the  Laplacian  image  processing 
suggested  by  Carver  Mead  by  including  the  dynamics  of  temporal  encoding  in  the  analysis  process. 

We  will  discuss  the  dynamic  neural  model  and  introduce  the  pulse  coupled  signal  used  as  an  information 
carrier.  We  examine  the  role  of  feedback  control  for  imaging  systems  using  intelligent  error  signals.  We 
show  how  control  signals  based  on  local  and  global  behavior  of  detectors  can  be  used  to  provide  a 
number  of  image  processing  technique  on  the  focal  plane.  These  include  edge  detection,  dynamic  range 
control,  segmentation  and  motion  detection.  We  examine  dynamic  range  control  using  adaptive 
histogram  processing  and  segmentation  by  adaptive  histogram  limiting  with  local  area  smoothing. 
Biological  mechanism  for  image  analysis  suggest  a  type  of  adaptive  histogram  processing  is  possible. 
VLSI  pulse  coupled  network  models,  using  an  integrate  and  fire  mechanism,  are  examined  for  fast 
generation  of  a  histogram  like  signals.  This  signal  is  used  to  control  the  gain  and  offset  parameters  of  the 
camera.  The  reverse  problem  is  examine  adding  local  receptor  field  for  image  segmentation. 
Conventional  image  pre-processing  usually  includes  histogram  equalization  to  provide  a  uniform 
distribution  of  the  pixel  information.  Histogram  equalization  at  the  sensor  would  maximize  the 
information  content  of  the  data  and  provide  uniform  imagery  for  robotic  control  applications.  An 
algorithm  is  developed,  which  allows  feedback  to  be  provided  to  the  image  collection  system. 

While  change  detection  algorithms  have  been  partially  successful,  many  important  task  remain 
unsolved.  We  suggest  solution  to  these  problems  may  lie  in  mimicking  architectures  and  algorithms 
used  by  biological  systems.  Pulsed  oscillatory  neural  network  design,  based  on  biomemonics,  is 
examined  for  application  to  these  these  problems  and  evaluated  for  application  to  image  analysis  and 
segmentation  of  multispectral  imagery  from  the  Satellite  Pour  I’Observation  de  la  Terre  (SPOT).  Using 
biological  systems  as  a  model  for  image  analysis  of  complex  data,  a  pulse  coupled  networks  using  an 


integrate  and  fire  mechanism  is  developed.  This  architecture,  based  on  layers  of  pulsed  coupled  neurons 
IS  tested  against  common  image  segmentation  problems.  Using  a  reset  activation  pulse  similar  to  that 
generated  by  sacatic  motor  commands,  an  algorithm  is  developed  which  demonstrates  the  biological 
vision  could  be  based  on  adaptive  histogram  techniques.  Histogram  processing  includes  dynamic  range 
control,  and  when  combined  with  smoothing  filters,  provides  segmentation. 

This  architecture  is  demonstrated  to  be  both  biologically  plausible  and  more  effective  than  conventional 
techniques.  Using  the  pulse  time-of-arrival  as  the  information  carrier,  the  image  is  reduced  to  a  time 
signal,  temporally  encoding  the  image,  which  allows  an  intelligent  filtering  using  hypothesis  feedback. 
This  technique  is  shown  to  be  uniquely  suited  to  some  multispectral/multisensor  imagery. 
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In  most  artificial  neural  networks,  a  synaptic  connection  between  two  units  performs  a  simple,  scalar 
weighting  of  the  information  transmitted.  In  real  nervous  systems,  synaptic  transmission  is  a  much  more 
complex  process  in  that  temporal  transformations  are  performed.  To  begin  with,  the  input  signals  are 
typically  encoded  in  pulse  trains,  the  pulses  producing  their  effects  on  the  postsynaptic  cell  with  some 
delay,  and  with  temporal  filtering  over  widely  varying  time  scales.  The  inter  action  between  different 
input  signals  impinging  upon  a  neuron  is  also  a  complex  affair,  depending  upon  electrical  properties  of 
the  postsynaptic  cell,  as  well  as  on  the  chemistry  of  receptors  and  second  messengers.  Leaving  aside 
these  chemical  mechanisms,  it  appears  that  many  important  aspects  of  neural  processing  might  be 
understood  in  terms  of  an  electrical  model  of  the  neuron  that  t^es  account,  the  dendritic  tree,  and  of  the 
disposition  of  synapses  upon  it  (Mel,  1994). 

In  order  to  study  patterns  of  synaptic  connection  on  the  dendritic  tree  that  could  give  rise  to 
discriminative  functions,  we  present  experimental  results  obtained  with  silicon  neurons,  either  singly  or 
in  small  networks.  VLSI  techniques  have  been  used  to  fabricate  neuromorphs  that  incorporate  the 
essential  features  of  the  "classical  neuron",  consisting  of  an  artificial  dendritic  tree,  and  a  leaky 
integrate-and-fire  "soma"  (Elias  and  Northmore,  1995).  The  dendritic  branches  are  composed  of  a  series 
of  identical  compartments  each  with  a  membrane  capacitance,  two  programmable  resistors  representing 
the  axial  cytoplasmic  and  membrane  resistances,  and  transistors  emulating  an  excitatory  and  an 
inhibitory  synapse.  The  excitatory  synapses  depolarize  the  compartment,  while  the  inhibitory  synapses 
hyperpolarize  it.  The  synapses,  when  activated  by  brief  pulses,  generate  impulse  responses  at  the  soma 
whose  amplitude  depends  upon  synapse  proximity.  (We  are  currently  developing  variable  conductance 
synapses).  Summation  of  these  impulse  responses  above  a  threshold  amplitude,  cause  the  soma  to 
generate  output  pulses,  which  can  be  routed,  via  high  speed  digital  circuitry,  to  an  arbitrary  set  of 
synapses  on  the  dendrites  of  the  same  and  other  neuromorphs  within  a  network.  A  range  of 
programmable  delays  can  be  imparted  to  each  pathway.  The  spike  firing  threshold  of  individual  somata 
can  be  controlled  by  impulse  activity  originating  within  the  network. 

The  neuromorphs  exhibit  the  classical  modes  of  signal  combination  by  temporal  and  spatial  summation. 
Thus  two  input  spikes  delivered  at  the  same  proximity  to  the  soma  but  on  different  dendritic  branches 
elicit  twice  the  amplitude  of  impulse  response  as  one  input  alone.  With  this  linear  mode  of  combination, 
adjustment  of  spike  firing  threshold  allows  a  neuromorph  to  respond  to  net  input  frequency  greater  than 
some  cutoff  value.  However,  when  two  impulses  activate  neighboring  synapses  close  together  in  time, 
their  effects  sum  sublinearly.  This  occurs  because  a  single  synaptic  activation  drives  the  compartmental 
potential  almost  to  the  driving  potential  for  the  synapse;  further  synaptic  activations  contiguous  in  space 
and  time  therefore  deliver  lesser  amounts  of  charge,  with  reduced  effects  accruing  to  the  potential 
appearing  at  the  spike  generator  of  the  soma.  Although  this  saturation  effect  is  readily  obtained  with  our 
artificial  synapses,  the  same  effects  may  well  occur  in  the  dendritic  branches  of  cortical  neurons  (Ferster 
and  Jagadeesh,  1992).  Whether  linear  or  sublinear  summation  of  synaptic  inputs  occurs  depends  to  a 


great  extent  up  on  the  spatial  arrangement  of  active  synapses  on  the  dendritic  tree.  Simultaneous 
activation  of  clusters  of  same-sign  synapses  promotes  sublinear  interactions,  while  juxtaposition  of 
opposite  sign,  or  dispersed  synaptic  sites  promotes  linear  interactions.  We  have  shown  that  synapse 
layout  can  be  used  for  spike  frequency  selectivity,  and  coincidence  detection  (Northmore  and  Elias, 
1996). 

It  seems  likely  that  these  properties  of  signal  interaction  on  the  dendritic  tree  could  be  exploited  for  the 
recognition  of  complex  spatio-temporal  patterns.  The  problem  is  how  to  find  the  appropriate  patterns  of 
connectivity  and  conduction  delays  to  perform  a  given  discrimination  task.  While  it  may  be  possible  in 
certain  cases  to  approach  the  problem  by  rearranging  synapses  to  emulate  weight  changes,  as  for 
artificial  neural  networks,  it  is  unlikely  that  the  full  capabilities  of  dendritic  processing  would  be  realized 
in  this  fashion.  Our  neuromorphic  system  allows  one  to  try  out  various  connections  schemes,  and 
procedures  for  generating  them,  ultimately  for  use  in  relatively  large  scale  networks. 

As  an  example  of  the  use  of  dendritic  tree  processing,  we  present  a  network  that  discriminates  pulse 
trains  representing  speech  sounds.  Vowel  sounds  were  digitized  and  converted  into  several  parallel 
impulse  trains,  as  follows.  A  pulse  in  a  given  train  was  generated  when  the  digitized  waveform  ascended 
across  a  fixed  threshold  level.  The  different  trains  representing  a  vowel  sound  activated  the  synapses  of 
a  set  of  4-branch  neuromorphs,  the  1st  stage  units,  which  in  turn  connected  with  a  set  of  output  units. 

The  objective  was  to  achieve  vowel  recognition  by  the  firing  of  the  appropriate  output  unit.  Connections 
from  each  input  train  to  the  different  1st  stage  units  were  selected  so  as  to  provide  a  partial 
discrimination  between  the  vowels.  Various  criteria  and  procedures  including  genetic  algorithms  and 
"development  rules"  are  being  tried.  The  connections  between  the  1st  and  output  layer  are  trained  using 
a  straightforward  delta  rule  involving  the  mapping  of  notional  weight  strengths  to  a  combination  of 
synapse  number  and  synapse  position  on  the  output  dendritic  trees. 

The  results  showed  consistent  discrimination  between  3  different  vowel  sounds.  The  auditory  systems  of 
animals  presumably  perform  similar  analyses  upon  spike  train  patterns,  particularly  with  low  sound 
frequencies,  or  low  frequencies  of  amplitude  modulation,  where  frequency  tuning  mechanisms  of  the  ear 
are  insufficiently  discriminative. 
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Several  years  ago,  when  we  designed  the  ETANN  chip,  analog  VLSI  appeared  to  be  the  only  feasible 
way  to  do  high  density  neural  computing.  In  the  last  five  years,  however,  digital  parallel  processing 
chips  capable  of  performing  neural  computation  functions  have  evolved  to  the  point  of  rough  equality 
with  analog  in  system  level  computational  density.  The  Naval  Air  Warfare  Center,  China  Lake  has 
developed  a  real  time,  hardware  and  software  system  designed  to  implement  and  evaluate  biologically 
inspired  retinal  and  cortical  models.  The  system  is  called  the  Missileborn  Artificial  Vision  System 
(MAVIS). 

The  hardware  is  based  on  the  Adaptive  Solutions  Inc.  massively  parallel  CNAPS  system  COHO  boards. 
Each  COHO  board  is  a  standard  size  6U  VME  card  featuring  4  (INAPS  chips  and  a  single  controller  in 
an  SIMD  configuration.  Each  CNAPS  chip  has  64  fixed  point,  RISC  processors  running  at  20  MHz 
giving  each  COHO  board  a  total  of  256  processors  with  a  theoretical  through  put  of  about  6  GOPS.  The 
MAVIS  system  is  designed  to  have  multiple  SIMD  machines  (MSIMD)  each  performing  a  different 
corticomorphic  functions  in  parallel.  The  current  system  has  three  COHO  boards  integrated  together  for 
a  total  of  768  processors  and  about  18  GOPS  of  capacity.  A  COHO  Companion  Board  is  built  to  support 
a  real  time  VSB  interface  to  an  imaging  seeker  and  a  NTSC  camera.  VME  is  used  to  communicate 
between  COHO  boards.  System  level  software  has  been  developed  which  allows  a  high  level  description 
of  corticomorphic  structures  to  be  translated  into  the  native  microcode  of  the  CNAPS  chips. 
Corticomorphic  structures  are  those  with  computational  forms  similar  to  the  neural  structures  of  the 
retina,  LGN  or  visual  cortex.  Most  standard  image  processing  forms  are  compatible  with  this  structure. 

The  MAVIS  system  is  a  real  time  hardware  and  software  system  designed  to  be  shrunk  into  a  volume 
compatible  with  air  launched  tactical  missiles.  Initial  versions  of  the  software  and  hardware  have  been 
completed  and  are  being  tested  with  various  sensors  including  a  missile  seeker.  We  will  discuss  results 
obtained  with  the  system  for  non-uniformity  correction  of  infrared  focal  plane  arrays  and  target 
detection  in  clutter. 


